Image processing apparatus, image processing method, and program

ABSTRACT

This invention more effectively suppresses color fringing in a color image by image processing. An image processing method includes estimating the degree of color fringing in a color image based on the color image that is generated by photo-electrically converting an object image and formed from a plurality of color planes. The method also includes removing from the color image the estimated degree of color fringing.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique of reducing color fringingof a color shot image.

2. Description of the Related Art

In a color image capturing system, a color which does not originallyexist appears as color fringing around a bright portion on an imageowing to chromatic aberration of an imaging optical system. Colorfringing readily occurs at a portion apart from the center wavelength ofthe imaging optical system. In a visible light color image capturingsystem, an artifact in blue, red, or purple as a mixture of blue and redappears as a fringe. This artifact is called color fringing or purplefringing.

Chromatic aberration can be optically suppressed to a certain degree bycombining lenses having different dispersions.

These days, as digital cameras are becoming compact, demands arise forincreasing the resolution of the image sensor and downsizing the opticalsystem. It becomes difficult to satisfactorily suppress chromaticaberration by only the optical system. Artifacts need to be reduced byimage processing.

Chromatic aberrations are roughly classified into transverse chromaticaberration (chromatic aberration of magnification) and longitudinalchromatic aberration (on-axis chromatic aberration). Transversechromatic aberration is a phenomenon that the image location shifts in adirection along the image plane depending on the wavelength, as shown inFIG. 1. Longitudinal chromatic aberration is a phenomenon that the imagelocation shifts in a direction along the optical axis depending on thewavelength, as shown in FIG. 2.

A digital image capturing system for the primary color system cancorrect transverse chromatic aberration by geometric transform of addingdifferent distortions to the R (Red), G (Green), and B (Blue) colorplanes, as disclosed in U.S. Pat. No. 6,724,702B1.

As for longitudinal chromatic aberration, for example, an image which isin focus on the G (Green) plane serving as the center wavelength of thevisible light region blurs on the R (Red) and B (Blue) planes serving asthe ends of the visible light region. Longitudinal chromatic aberrationcannot be corrected by geometric transform, unlike transverse chromaticaberration. Hence, there is proposed a method of correcting longitudinalchromatic aberration by adding different edge enhancement processes tothe R, G, and B planes, as disclosed in Japanese Patent Laid-Open No.2003-018407. Further, there is proposed a method of making longitudinalchromatic aberration less conspicuous by decreasing chroma in a regionwhere color fringing occurs, as disclosed in Japanese Patent Laid-OpenNo. 2001-145117.

However, deconvolution and approximate edge enhancement processing asdescribed in Japanese Patent Laid-Open No. 2003-018407 cannot obtain aproper result unless an accurate point spread function is known. In animage capturing apparatus such as a camera in which the object distanceand shooting conditions change, the state of the optical systemincluding the zoom position, F-number, and focus position also changes.It is difficult to obtain an accurate point spread function.Deconvolution can be used in only the linear reaction boundary of theimage sensor, and cannot reduce color fringing around saturated pixels.

The optical system of a general color image capturing apparatusoptically corrects chromatic aberration to a certain degree, and colorfringing hardly stands out in a normal brightness range. Instead, whenan excessively bright object exists within the frame to saturate pixels,a small quantity of leakage light which cannot be completely correctedoften acts as a significant quantity and causes color fringing. That is,the technique described in Japanese Patent Laid-Open No. 2003-018407cannot correct this color fringing.

As described in Japanese Patent Laid-Open No. 2001-145117, the processto decrease chroma can cancel a fringing color and reduce unnaturalness.However, the original object color is also influenced by this processand becomes grayish regardless of the presence/absence of colorfringing.

SUMMARY OF THE INVENTION

It is desirable to overcome the conventional drawbacks, and to moreeffectively suppress color fringing in a color image by imageprocessing.

According to a first aspect of the present invention there is providedan image processing apparatus, for processing original image datarepresenting a captured color image that has been subjected tophotoelectric conversion, comprising estimation unit which generatesestimated color fringing information representing an estimate of colorfringing in the captured image on the basis of the original image data,and removing unit which employs the estimated color fringing informationto generate modified image data based on the original image data so asto compensate for such color fringing in at least a part of the capturedimage.

According to a second aspect of the present invention there is provideda method of processing original image data representing a captured colorimage that has been subjected to photoelectric conversion, the methodcomprising the steps of generating estimated color fringing informationrepresenting an estimate of color fringing in the captured color imageon the basis of the original image data, and employing the estimatedcolor fringing information to generate modified image data based on theoriginal image data so as to compensate for such color fringing in atleast a part of the captured image.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing a transverse chromatic aberration generationprinciple;

FIG. 2 is a view showing a longitudinal chromatic aberration generationprinciple;

FIG. 3 is a view showing the arrangement of a color image capturingapparatus to which an image processing method according to the firstembodiment is applicable;

FIG. 4 is a graph showing the spectral transmittance of a primary colorfilter;

FIG. 5 is a view showing the layout of color elements of the primarycolor filter;

FIG. 6 is a graph showing blue fringing at the boundary between brightand dark regions;

FIG. 7 is a flowchart showing a color fringing removing operation byimage processing in the color image capturing apparatus according to thefirst embodiment;

FIG. 8 is a graph showing the typical profiles of the B and G planes ofa bright object;

FIG. 9A is a view showing a distance from a saturated pixel to aperipheral pixel;

FIG. 9B is a view showing a distance from a peripheral pixel to asaturated pixel;

FIG. 10 is a graph showing chromaticity coordinates;

FIG. 11 is a view showing the arrangement of a color image capturingapparatus to which an image processing method according to the secondembodiment is applicable;

FIG. 12 is a flowchart showing a color fringing removing operation byimage processing in the color image capturing apparatus according to thesecond embodiment;

FIG. 13 is a graph showing a non-linear conversion characteristic;

FIG. 14 is a graph showing a saturation degree profile;

FIG. 15 is a graph showing a convolution kernel;

FIG. 16 is a graph showing a convolution result;

FIG. 17 is a graph showing the chromaticity coordinates of the U-Vplane;

FIG. 18 is a view showing the arrangement of a color image capturingapparatus to which an image processing method according to the thirdembodiment is applicable;

FIG. 19 is a flowchart showing a color fringing removing operation byimage processing in the color image capturing apparatus according to thethird embodiment;

FIG. 20 is a graph showing the typical intensity profile of redfringing;

FIG. 21 is a graph showing a non-linear conversion characteristic; and

FIG. 22 is a graph showing an excessive removal suppressing principle.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will be described below.

First, an outline of the embodiments of the present invention will beexplained.

Embodiments of the present invention can provide an image processingapparatus, method, and program capable of effectively removing colorfringing around saturated pixels of a color image which is shot by acolor image capturing apparatus and suffers color fringing, andreproducing the original color.

The image processing apparatus according to one embodiment of thepresent invention comprises an estimation unit which estimates thedegree of color fringing from image data representing a color image thatis shot by the image capturing system and made up of a plurality ofcolor planes, and a removing unit which generates modified image datarepresenting a color image corresponding to the original image but withreduced color fringing. For example, the removing unit may generate themodified image data for a pixel by subtracting the estimated amount fromthe original image data for the pixel concerned. Of course, instead ofsubtraction, any other suitable kind of operation can be performed togenerate the modified image data from the original image data usinginformation about the color fringing estimated by the estimation unit.For example, it would be possible to multiply the original image datafor a pixel by a correction factor dependent on the estimated colorfringing information. Preferably, the removing unit sets, as a removaltarget, a color plane exhibiting the intensity of a wavelength rangewhere chromatic aberration remains in the imaging optical system used inimage shooting. The removing unit subtracts the degree of fringing onthe target color plane.

The image processing apparatus according to one embodiment of thepresent invention further comprises a region determination unit, and theestimation unit can also employ different estimation methods dependingon the type determined by the region determination unit. The imageprocessing apparatus preferably further comprises an excessive removalsuppressing unit, and the excessive removal suppressing unit can alsosuppress change of the hue of a color image that is caused by theremoving unit.

In some embodiments the color plane subjected to removal is the R (Red)or B (Blue) plane, but can also be a color difference plane U/Vrepresenting a color tincture corresponding to the R or B plane.

Since color fringing is a spatial action, it is preferable to provide aspatial calculation unit which executes spatial calculation in order toestimate the image intensity of color fringing from a color image. Thespatial calculation can be applied not only to a color plane subjectedto removal, but also to a color plane having high resolution as areference plane. The reference plane is a color plane having awavelength range or luminance at which chromatic aberration issatisfactorily corrected in the imaging optical system used in imageshooting. The reference plane is generally the G (Green) or Y(luminance) plane. Several kinds of spatial calculation are conceivable.One of them is (1) image distance transform of calculating a distance toeach pixel from a saturated region where fringing occurs. The saturatedregion is a saturated pixel on the reference plane or the color planesubjected to removal.

Alternatively, (2) convolution processing is used. In this case, priorto convolution, the image intensity of a reference plane undergoesnon-linear conversion in order to correct non-linearity upon saturation.The intensity of the reference plane abruptly increases near a regionwhere this intensity is saturated. The non-linear conversion can also beconversion into two values representing whether the intensity of thereference plane is saturated. The convolution processing is done for theimage intensity having undergone the non-linear conversion.

The kernel of convolution processing preferably simulates a decrease inthe resolution of the imaging optical system, and is, for example, thePSF (Point-Spread Function) of the imaging optical system at the typicalwavelength of a color band corresponding to a plane subjected toremoval. Since the PSF of the imaging optical system changes dependingon the image location and the state of the optical system in shooting,the convolution kernel is desirably changed in accordance with thesefactors. However, it is also possible to adopt a kernel enveloping aplurality of PSFs of the imaging optical system that change inaccordance with the image location and the state of the optical systemin shooting.

The PSF also changes depending on the wavelength within even a singlecolor band though this cannot be known. Thus, it is also possible toadopt a kernel enveloping PSFs which are different in accordance with aplurality of wavelengths within a color band corresponding to a planesubjected to removal. This kernel desirably decreases in accordance withthe distance from the center. For a simple expression, it is practicalto adopt an axially symmetrical kernel or define the kernel by anexponential function or Gaussian function.

As spatial calculation, (3) image intensity gradient calculation is alsoavailable. The image intensity gradient is the intensity gradient of thereference plane or the color plane subjected to removal.

When the spatial calculation unit executes (1) image distance transform,the estimation unit outputs, in a region where the color plane subjectedto removal is saturated, a value which increases in a direction awayfrom a region where the reference color plane is saturated. In a regionwhere the color plane subjected to removal is not saturated, theestimation unit outputs a value which decreases in a direction away froma region where the color plane subjected to removal is saturated. Theregion determination unit switches the estimation method. These outputsincrease as the difference between the saturation radii of the colorplane subjected to removal and the reference color plane increases. Theratio of increase/decrease effectively changes in accordance with theimage height and the state of the imaging optical system used to shootan image.

When the spatial calculation unit executes (2) convolution processing or(3) image intensity gradient calculation, the estimation unit outputs avalue depending on the convolution value or image intensity gradient. Inthis case, the estimation unit can simply output a value proportional tothe convolution value or image intensity gradient.

The region determination unit preferably determines whether the colorplane subjected to removal is saturated. Based on the determination, theregion determination unit switches the estimation method of theestimation unit, selects one of estimated values obtained by theestimation unit, or interpolates estimated values obtained by theestimation unit.

In this manner, the estimated value of the fringing amount to be removedcan be attained. However, this estimated value is not always accurate,and may be large or small. If the estimated value is small, colorfringing cannot be completely removed and remains slightly. If theestimated value is large, color fringing is excessively removed and thehue of color fringing is inverted. According to experiments by thepresent inventors, the latter excessive removal results in a highlyunnatural image in comparison with the former removal shortage. Tosuppress inversion of the hue, an excessive removal suppressing unit ispreferably arranged to set only pixels in the color gamut of apredetermined region as removal targets of the removing unit, and/orrestrict even the color gamut after the change to a predeterminedregion. These two predetermined regions may be the same. The color gamutand the predetermined region within it may be analysed by converting theimage data in an original color coordinate system (e.g. an RGB system)into another color co-ordinate system (e.g. the Lab system or the LUVsystem), and then selecting only pixels in certain regions (e.g.quadrants) of that other color coordinate system. Alternatively, thepart of the color plane subjected to removal can be set to a regionwhere its intensity is higher than that of the reference color plane. Asa method of suppressing change of the hue, change of the hue angle bythe removing unit may also be suppressed to a predetermined angularrange in a suitable color co-ordinate system.

The above-described image processing apparatus can provide an image inwhich color fringing is reduced.

A color image capturing apparatus having the image processing apparatusaccording to the embodiments of the present invention suffices tocomprise an optical system which suppresses chromatic aberration in onlya wavelength range corresponding to at least one color plane. Thechromatic aberration restriction on the optical system can be eased forthe remaining color planes.

Generally, an imaging optical system mounted in a color image capturingapparatus performs predetermined chromatic aberration correction basedon the trade-off between the size, the cost, and various aberrationcorrections. The embodiments of the present invention can ease therestriction on chromatic aberration. As a result, the imaging opticalsystem can achieve other aberration corrections, downsizing, and costreduction at higher level.

Embodiments of the present invention will be described in detail below.

First Embodiment

An image processing method according to the first embodiment of thepresent invention will be described with reference to the accompanyingdrawings. The same reference numerals denote the same parts throughoutthe drawings.

FIG. 3 shows an example of a color image capturing apparatus 100 towhich the image processing method according to the first embodiment isapplicable.

The color image capturing apparatus 100 comprises an imaging opticalsystem 110, image sensor 120, A/D conversion unit 130, demosaicking unit140, color fringing removing unit 150, vision correction unit 160,compression unit 170, and recording unit 180. Note that a field (object)f, and R (Red), G (Green), and B (Blue) rays shown in FIG. 2 are notbuilding components of the color image capturing apparatus 100, but areshown in FIG. 3 for descriptive convenience.

In FIG. 3, the field f is imaged via the imaging optical system 110 onthe image sensor 120 which photo-electrically converts an object image.Generally, an imaging optical system mounted in a color image capturingapparatus performs predetermined chromatic aberration correction basedon the trade-off between the size, the cost, and various aberrationcorrections. However, the imaging optical system 110 according to thefirst embodiment sufficiently corrects longitudinal chromatic aberrationin only the R and G wavelength ranges, and longitudinal chromaticaberration remains in the B band. Since the trade-off restriction“longitudinal chromatic aberration in the B band” is excluded, otheraberration corrections, downsizing, and cost reduction can be achievedat higher level.

The image sensor 120 is a one-CCD color image sensor having a generalprimary color filter. As shown in FIG. 4, the primary color filter ismade up of three types of color filters having dominant transmissionwavebands around 650 nm, 550 nm, and 450 nm, respectively. These colorfilters sense color planes corresponding to the R (Red), G (Green), andB (Blue) bands. In the one-CCD color image sensor, the color filters arespatially arrayed for respective pixels as shown in FIG. 5, and eachpixel can obtain only an intensity on a single color plane. For thisreason, the image sensor outputs a color mosaic image.

The A/D conversion unit 130 converts a color mosaic image output as ananalog voltage from the image sensor into digital data suited tosubsequent image processing.

The demosaicking unit 140 interpolates a color mosaic image, generatinga color image having information of the R, G, and B colors in allpixels. As the interpolation method, many methods are proposed includingsimple linear interpolation and a complicated method as described in “E.Chang, S. Cheung, and D. Pan, “Color filter array recovery using athreshold-based variable number of gradients,” Proc. SPIE, vol. 3650,pp. 36-43, January 1999”. However, the present invention does not limitthe interpolation method.

The resolution of the B plane of the generated color image is lower thanthose of the R and G planes owing to chromatic aberration of the imagingoptical system 110. At the boundary between bright and dark regions,blue blurs as shown in FIG. 6, and an artifact like a blue fringeappears around the bright region.

In the above description, the image sensor 120 has R, G, and B primarycolor filters. However, even complementary color filters can obtain acolor image similarly made up of the R, G, and B color planes by colorconversion processing.

The color fringing removing unit 150 removes the blue artifact from acolor image by image processing. The image processing method accordingto the first embodiment is directed to this removing processing, whichwill be explained in detail later.

Then, the vision correction unit 160 executes processing. The visioncorrection unit 160 processes a color image in order to improve mainlythe image appearance. For example, the vision correction unit 160performs image corrections such as tone curve (gamma) correction, chromaenhancement, hue correction, and edge enhancement.

At the final stage of the processing, the compression unit 170compresses a corrected color image by JPEG or the like to decrease therecording size.

In practice, the building components from the image sensor 120 to therecording unit 180 are not always separate devices. A singlemicroprocessor may perform processes corresponding to a plurality ofbuilding components.

The recording unit 180 records a processed digital image signal on arecording medium such as a flash memory.

FIG. 7 is a flowchart showing a color fringing removing operation byimage processing in the color image capturing apparatus 100 having thisarrangement. In this embodiment, the steps in the flowchart of FIG. 7may be carried out by a processor of the apparatus 100, such as a CPU(not shown), which executes a program (not shown) stored in a memory(not shown) of the apparatus. The same is true for other embodimentsdescribed later.

As shown in FIG. 7, the process of the color fringing removing unit 150includes spatial calculation step S151, region determination step S152,estimation step S153, excessive removal suppressing step S154, andremoving step S155.

In the color fringing removing operation, the B plane is set as aremoval target, and the G plane is used as a reference plane.

FIG. 8 shows the typical profiles of the B and G planes of a brightobject.

In FIG. 8, the abscissa axis represents a section on an image, and theordinate axis represents the intensities of the B and G planes. In FIG.8, a bright object exceeding the saturation brightness exists at thecenter. The bottom of the profile spreads at the periphery, which is notbright originally, of the bright object by light leaking from the brightobject owing to aberration or flare. The degree of fringing depends onthe brightness of the bright object, and decreases exponentially in adirection away from the bright object. Even the G plane suffersfringing, and its profile spreads to a certain degree though the spreadis smaller than that of the B plane. An intensity at a predeterminedsaturation level or higher cannot be measured by the image sensor, andis rounded down. In a shot image, both the G and B planes are saturatedat an intensity much higher than that of the original bright object,generating a white saturated region. Note that the R plane has the sameprofile as that of the G plane. After the saturation, the intensity ofthe G plane attenuates. However, since the saturation radius of the Bplane is larger, the image intensity difference between the G and Bplanes becomes larger, and the saturated region becomes light bluish.When the B plane reaches its saturation radius, its intensity alsostarts attenuating, decreasing the image intensity difference betweenthe G and B planes. After the G plane reaches the bottom of its profile,only the B plane has an intensity and the fringing becomes a deep blue.

In the object image, light blue fringing and deep blue fringing arerecognized as unnatural blue fringing. If the degree of B fringing isalmost equal to that of G fringing, the fringing is recognized as thecolor of a bright object and becomes natural. Such fringing is aneffective image representation expressing the brightness of a brightobject exceeding the saturation brightness.

From this, in spatial calculation step S151, a saturated pixel regionwhere the intensities of both the G and B color planes are equal to orhigher than a predetermined threshold is extracted. Distances d_(G) andd_(B) of pixels from the saturated pixel region are calculated as pixelwidths. For example, assuming that hatched cells in FIG. 9A aresaturated pixels, distances are represented by numerical values in thepixels.

The threshold is set to an output value at which the output value of theA/D conversion unit and the incident light intensity lose a proportionalrelationship. A state in which a pixel has a higher output value will becalled saturation. Distance calculation is generally called imagedistance transform. This distance is not limited to an accurateEuclidean distance, but may also be a quasi-Euclidean distance,chessboard distance, or city block distance. By the image distancetransform, a saturated pixel takes 0, and an unsaturated pixel takes apositive value. By the transform of the B plane, pixels are classifiedinto a region A1 where B is saturated, and a region A2 where B is notsaturated (see FIG. 8).

In this step, at least a distance d_(nB) to the region A1 from a regionwhere B is not saturated is calculated in the same way. d_(B) isrepresented as shown in FIG. 9A, whereas d_(nB) is represented as shownin FIG. 9B. Note that d_(nB) can also be represented as a negative valueof d_(B).

In region determination step S152, processing targets are distributed toa plurality of processing methods S153 a and S153 b in estimation stepS153 in accordance with the calculation result. In the first embodiment,the region A1 where B is saturated is assigned to S153 a, and the regionA2 where B is not saturated is assigned to S153 b.

In estimation step S153, the intensity of the extra B plane whichgenerates color fringing is estimated for each pixel of the color image.The estimation method changes depending on whether B is saturated.Estimation calculation is executed in S153 a and S153 b, whichcorrespond to the regions A1 and A2 in FIG. 8, respectively.

As described above, the image intensity of the B plane subjected toremoval is the difference between the B and G planes. The estimatedamount increases in the region A1 in a direction away from the regionwhere G is saturated, and decreases in the region A2 in a direction awayfrom the region A1.

The estimated amount depends on the brightness of a bright object, butcannot be obtained directly due to saturation. Instead, the firstembodiment adopts the difference “d_(G)−d_(B)” or “d_(G)+d_(nB)” betweenthe B and G saturation radii. In step S153 a for the region A1, anestimated fringing amount E is calculated as the estimated amount by

E=(k1(d _(G) +d _(nB))+k0)×d _(G)/(d _(G) +d _(nB))

In step S153 b for the region A2, the estimated fringing amount E iscalculated by

E=(k1(d _(G) −d _(B))+k0)exp(−k2d _(B))

Then, the estimated fringing amounts E are transferred to excessiveremoval suppressing step S154. At this time, no Mach band appearsbecause the estimated fringing amount E coincides with E0=(k1d_(g)+k0)on the boundary between the regions A1 and A2.

k0, k1, and k2 are constants, and change depending on the imagingoptical system and the pixel pitch of the image sensor. It is desirableto obtain values suitable for approximating the fringing amount from ashot image.

Strictly speaking, the characteristics of the imaging optical systemchange depending on the image height and states such as the zoomposition, F-number, focus position, and lens interchange. It is alsoeffective to change the constants k0, k1, and k2 in accordance withthese factors. Considering the presence of the excessive removalsuppressing step to be described below, it is desirable to set constantsfor estimating a fringing amount larger than a value suitable forapproximating the fringing amount. Alternatively, considering thepresence of the excessive removal suppressing step to be describedbelow, it is also possible to set constants for estimating a largefringing amount so as to cope with change of the characteristic of theimaging optical system.

In excessive removal suppressing step S154, the estimated amount E iscorrected to obtain an actual removal amount E′. The removal amountestimated in step S153 complies with a predetermined model, and does notcoincide with an actual fringing amount. For example, even lightdetected on the B plane exhibits different fringes between light havinga wavelength of 450 nm and that having a wavelength of 400 nm. However,this is not considered in step S153. If the removal amount isexcessively small, the image remains slightly bluish even after removal.If the removal amount is excessively large, B is excessively removedagainst the gray background, and the image becomes yellowish green. Inthe latter case, the user feels that the image is unnatural. In thisstep, therefore, fringing removal is restricted within a predeterminedhue range. For this purpose, the chromaticity of pixels is calculatedfirst. For a given pixel, a device-dependent conversion is carried outto convert the intensities of the pixel in the RGB color space (R, G,and B planes) into intensities in the LAB color space,

$\begin{matrix}{{\begin{pmatrix}x \\y \\z\end{pmatrix} = {\begin{pmatrix}0.41 & 0.36 & 0.18 \\0.21 & 0.75 & 0.07 \\0.02 & 0.12 & 0.95\end{pmatrix}\begin{pmatrix}R \\G \\B\end{pmatrix}}}{a = {5\left( {x - y} \right)}}{b = {2\left( {y - 1} \right)}}} & (1)\end{matrix}$

where R is the image intensity of the pixel in the R color plane, G isthe image intensity of the pixel in the G color plane, B is the imageintensity of the pixel in the B color plane, x, y and z are colorcoordinates of the pixel in the CIE XYZ color space, a and b areco-ordinates of the pixel in color-opponent dimensions of the CIE LABcolor space, and the matrix (0.41 . . . 0.95) is an exemplary conversionmatrix dependent on the particular characteristics of the components ofthe imaging system. The values in the matrix can be changed according tothe particular optical characteristics of the apparatus.

FIG. 10 shows the a-b plane of chromaticity coordinates. Blue fallswithin the fourth quadrant. When the estimated amount E is removed fromthe B intensity, the chromaticity moves to the upper left as indicatedby dotted arrows. The start point of the arrow represents a chromaticitybefore removal, and its tip represents a chromaticity after removal.From this, the hue range where fringing is removed is restricted to a>0and b<0.

Two conditions, B>0.22R+0.72G and B>−1.82R+3.54G, are used when settinga removal amount E′ in this embodiment. In step S154, E′=0 for pixelswhich do not meet these conditions, and as a result these pixels areexcluded from being removal targets. These pixels do not change inremoving step S155, and their pixel values are not influenced by colorfringing removing step S155. In other words, only pixels which meet atleast one of the two conditions are removal targets.

The removal amount for pixels which meet these conditions is set to

E′=min(E,B−(0.22R+0.72G),B−(−1.82R+3.54G))

Then, E′ is transferred to removing step S155. As indicated by solidarrows in FIG. 10, the chromaticity change by removal of E′ isrestricted within the fourth quadrant (a>0 and b<0). B can be preventedfrom decreasing more than the hue restriction range in the removingstep.

In removing step S155, a new intensity of the B plane is calculated bysubtracting the removal amount E′ from the intensity of the B plane:

B=B−E′

A color image whose B plane is corrected is transferred as an outputfrom the color fringing removing unit 150 to the vision correction unit160.

The first embodiment has described a color image capturing apparatushaving the imaging optical system 110 to recording unit 180. It is alsopossible to constitute an image processing apparatus which performs onlycolor fringing removal by arranging some or all units except for thecolor fringing removing unit 150 as separate devices. In this case, animage processing apparatus is arranged separately from a color imagecapturing apparatus. The image processing apparatus is configured toread (receive) a color image which is shot by the color image capturingapparatus and recorded on a recording medium such as a semiconductormemory or magnetic/optical disk.

The color image capturing system having the blue fringing removing unitaccording to the first embodiment can effectively remove blue fringingand obtain a natural shot image. The accessory imaging optical systemcan ease the restriction on longitudinal chromatic aberration in the Bband. Other aberration corrections, downsizing, and cost reduction canbe achieved on a higher level.

Second Embodiment

FIG. 11 shows an example of a color image capturing apparatus 200 towhich an image processing method according to the second embodiment ofthe present invention is applicable. In FIG. 11, the same referencenumerals as those in FIG. 3 denote the same functional parts.

The color image capturing apparatus 200 comprises an imaging opticalsystem 210, a color separation prism 215, image sensors 220, an A/Dconversion unit 230, a color conversion unit 235, a color fringingremoving unit 250, a vision correction unit 160, a compression unit 170,and a recording unit 180.

The image sensors 220 according to the second embodiment are three CCDimage sensors, unlike the first embodiment. The color separation prism215 is added for the image sensors 220, and the demosaicking unit 140 inthe first embodiment is omitted.

In FIG. 11, rays traveling from an object are formed into images on theimage sensors 220 via the imaging optical system 210 and colorseparation prism 215. In the color separation prism 215, the propagationdirection of light changes depending on the wavelength of light. Thus,rays having different R (Red), G (Green), and B (Blue) wavelength rangesreach the different image sensors 220. For this reason, the imagesensors 220 do not have any color filter, and obtain imagescorresponding to the R, G, and B color planes.

The imaging optical system 210 according to the second embodimentsufficiently corrects longitudinal chromatic aberration in only thewavelength range within the G band, and longitudinal chromaticaberration remains in the R and B bands. The three CCD image sensors canalso correct longitudinal chromatic aberration by adjusting theirhorizontal positions. However, this adjustment is not considered in thesecond embodiment because it cannot cope with variations in aberrationamount caused by the zoom position of the optical system or the like. Asa result, the resolutions of the R and B planes are lower than that ofthe G plane. Red and blue blur at the boundary between bright and darkregions in a color image obtained by compositing the three planes. Anartifact like a red, blue, or purple fringe appears around the brightregion.

The A/D conversion unit 230 converts images of the R, G, and B colorplanes output as analog voltages from the three CCD image sensors 220into digital data suited to subsequent image processing.

The color conversion unit 235 converts the color representation from RGBinto YUV. The color conversion unit 235 uses matrix operation, obtainingthe three, Y, U, and V planes:

$\begin{matrix}{\begin{pmatrix}Y \\U \\V\end{pmatrix} = {\begin{pmatrix}0.30 & 0.59 & 0.11 \\{- 0.15} & {- 0.29} & 0.44 \\0.61 & {- 0.52} & 0.10\end{pmatrix}\begin{pmatrix}R \\G \\B\end{pmatrix}}} & (2)\end{matrix}$

Y represents brightness, U represents blueness, and V representsredness.

The color fringing removing unit 250 removes the artifact from a colorimage by image processing. The image processing method according to thesecond embodiment is directed to this removing processing, which will beexplained in detail later.

The vision correction unit 160, compression unit 170, and recording unit180 are identical to those in the first embodiment.

FIG. 12 is a flowchart showing a color fringing removing operation byimage processing in the color image capturing apparatus 200 having thisarrangement.

As shown in FIG. 12, the process of the color fringing removing unit 250includes a spatial calculation step S251, estimation step S253,excessive removal suppressing step S254, and removing step S255. Thecolor fringing removing unit 250 sets the R and B planes as removaltargets, and uses the Y plane as a reference plane.

In the spatial calculation step S251, a degree S of saturation of eachpixel is calculated by executing non-linear conversion for the intensityof the Y plane, and convolution processing for the degree S ofsaturation is performed. The non-linear conversion corrects thebrightness of a bright object that is represented excessively low owingto saturation. As a result of the conversion, as shown in FIG. 13, the Yintensity abruptly increases near a region where the Y intensity issaturated, and exhibits a larger value as compared with the proportionalrelationship between the Y intensity and the degree of saturation in theunsaturated region. In FIG. 13, the maximum value is 4. In the followingdescription, the Y intensity is so normalized as to set the maximumvalue of the degree of saturation to 1. The degree S of saturation has aprofile indicated by a solid line in FIG. 14, and the Y intensity has aprofile indicated by a dotted line in FIG. 14. In FIG. 14 the abscissaaxis represents pixel position and the ordinate axis representsintensity.

As a simple example of the non-linear conversion, the Y intensity afterconversion may take one of two values, for example, 1 near the regionwhere the Y intensity is saturated (e.g., Y>0.8), and 0 in theunsaturated region (Y<0.8).

Convolution processing is done for the degree S of saturation, obtainingconvolution results S_(R) and S_(B). Since S_(R) and S_(B) can beconsidered as fringes in the B (Blue) and R (Red) bands, twocorresponding convolution processes are executed:

S_(R)=k_(R)

S

S_(B)=k_(B)

S  (3)

FIG. 15 shows an example of the convolution kernel k_(R) and FIG. 16shows an example of the convolution result S_(R) for the degree S ofsaturation having the profile shown in FIG. 14. In FIGS. 15 and 16 theabscissa axis represents pixel position and the ordinate axis representsintensity.

The convolution kernels k_(R) and k_(B) simulate a decrease in theresolution of the imaging optical system 210, and can use, for example,the PSFs (Point-Spread Functions) of typical wavelengths in the R and Bbands. An effective example of the typical wavelength in the B band isthe mercury lamp emission line (405 nm) often present in night scenes.The characteristics of the PSF and imaging optical system changedepending on the image location and lens states such as the zoomposition, F-number, focus position, and lens interchange. Theconvolution kernels are desirably changed in accordance with thesefactors.

Alternatively, considering the presence of excessive removal suppressingstep S254 to be described below, it is also possible to set aconvolution kernel which envelops a plurality of changeable PSFs andestimates an excessively large fringing amount so as to cope with changein the characteristics of the imaging optical system. Convolution usingdifferent kernels depending on the image location puts a heavycalculation load. Hence, the calculation load can be effectively reducedusing an axially symmetrical convolution kernel which envelops change ofthe PSFs in the image direction within the entire image plane or apredetermined region of the image plane. The calculation load can alsobe effectively reduced using a shift invariant convolution kernel whichenvelops even change depending on the image height. At this time, theconvolution kernel may be an exponential function or Gaussian function.

Similarly, a convolution kernel which envelops a plurality of PSFs thatchange depending on the wavelengths in the R and B bands may also beset. Particularly in this case, the B band effectively includes themercury lamp emission line (405 nm).

These convolution kernels desirably decrease in intensity as thedistance from a center position increases.

In estimation step S253, estimated values EU and EV of the degree ofcolor fringing in the U and V planes of the CIE LUV color space aregenerated. In this case, EU and EV are simply constant multiples ofS_(R) and S_(B) respectively:

EU=0.424fB·S _(B)

EV=0.877fR·S _(R)

where fB and fR correspond to original image intensities in thesaturated region in the B and R color planes respectively, and take avalue of 1 to 10. For example, based on empirical studies fB and fR canbe set to 4 to obtain satisfactory results.

In excessive removal suppressing step S254, the estimated amounts EU andEV are corrected to obtain actual removal amounts E′ in the U and Vplanes. Attention is paid to the chromaticity coordinates, similar tothe first embodiment. FIG. 17 shows the chromaticity coordinates of theU-V plane. Blue is U>0, and red is V>0. When EU and EV are removed fromthe U and V intensities (i.e. if step S254 is omitted), the chromaticitymoves to the lower left as indicated by dotted arrows. The movingdirection changes depending on the ratio of EU and EV. The start pointof the arrow represents a chromaticity before removal, and its tiprepresents a chromaticity after removal of the estimated amounts EU andEV.

In this embodiment, step S254 restricts the hue restriction range to arange defined by U>0 and V>0 (i.e. the first, second and fourthquadrants but not the third quadrant). EU′=0 for pixels having U≦0, andEV′=0 for pixels having V≦0. For pixels having U>0,

EU′=min(EU,U)

For pixels having V>0,

EV′=min(EV,V)

Then, EU′ and EV′ are transferred to removing step S255. As indicated bysolid arrows in FIG. 17, the chromaticity change by removal of EU′ andEV′ is restricted within the corresponding quadrants. Further, only Vchanges in the second quadrant, only U changes in the fourth quadrant,and neither of them changes in the third quadrant. In terms of changesto R and B, this means that the R and B intensities do not lower fromthe luminance Y, and R and B intensities originally lower than Y do notchange. Referring to FIG. 17, in the first quadrant, the solid arrows(representing the actual changes) coincide with the dotted arrows insome cases but not others. The cases where they coincide are the oneswhich remain in the first quadrant after the change. In the remainingcases, the dotted arrow ends outside the first quadrant. In this case,for dotted arrows which end in the fourth quadrant, the change isrestricted so that V=0 and U for the head of the solid arrow is the sameas for the dotted arrow. For dotted arrows which end in the secondquadrant, the change is restricted so that U=0 and V for the head of thesolid arrow is the same as for the dotted arrow. For dotted arrows whichend in the third quadrant, the change is restricted so that U and V forthe head of the solid arrow are both 0. In the second quadrant, nochange to U is permitted, so the solid arrows all extend vertically. Vfor the head of each solid arrow is the same as for the correspondingdotted arrow, unless the dotted arrow ends outside the second quadrant(V<0), in which case V is set to 0. In the third quadrant, no changes atall are permitted, so no solid lines are shown in FIG. 17. In the fourthquadrant, no change to V is permitted, so the solid arrows all extendhorizontally. U for the head of each solid arrow is the same as for thecorresponding dotted arrow, unless the dotted arrow ends outside thefourth quadrant (U<0), in which case U is set to 0.

In removing step S255, new U and V plane values are calculated bysubtracting the removal amounts EU′ and EV′ from the U and V planevalues:

U=U−EU′

V=V−EV′

A color image whose U and V planes are corrected is transferred as anoutput from the color fringing removing unit 250 to the visioncorrection unit 160.

In the second embodiment, the fringes of the B and R planes are mixed inthe Y plane. The amount of white fringing is slightly larger than thatin a case where the G plane is used as a reference plane. However, thecost of the processing apparatus can be suppressed by executing maincalculation on the U and V planes which do not require high precision.The optical system in the second embodiment needs to have highresolution in only the G band, and the restriction on chromaticaberration in the R and B bands can be eased.

Third Embodiment

FIG. 18 shows an example of a color image capturing apparatus 300 towhich an image processing method according to the third embodiment ofthe present invention is applicable. In FIG. 18, the same referencenumerals as those in FIG. 3 denote the same functional parts.

The color image capturing apparatus 300 comprises an imaging opticalsystem 310, image sensor 120, A/D conversion unit 130, demosaicking unit140, color fringing removing unit 350, vision correction unit 160,compression unit 170, and recording unit 180.

The imaging optical system 310 according to the third embodiment formslight traveling from an object into an image on the image sensor 120.Longitudinal chromatic aberration is sufficiently corrected by theimaging optical system 310 in the wavelength range of the G and B bands,but remains in the R band.

The following phenomenon occurs on the R, G, and B planes of a colorimage which is formed on the imaging optical system 310 and generatedvia the image sensor 120, A/D conversion unit 130, and demosaicking unit140. That is, the resolution of the R plane is lower than those of the Gand B planes under the influence of chromatic aberration of the imagingoptical system 310. Red blurs at the boundary between bright and darkregions in a color image obtained by compositing the three planes. Anartifact like a red fringe appears around the bright region.

The color fringing removing unit 350 removes the red artifact from acolor image by image processing. The image processing method accordingto the third embodiment concerns this removing processing, which will beexplained in detail later.

The image sensor 120, A/D conversion unit 130, demosaicking unit 140,vision correction unit 160, compression unit 170, and recording unit 180are identical to those in the first embodiment.

FIG. 19 is a flowchart showing a color fringing removing operation byimage processing in the color image capturing apparatus 300 having thisarrangement.

As shown in FIG. 19, the process of the color fringing removing unit 350includes spatial calculation step S351, estimation step S353, regiondetermination step S352, excessive removal suppressing step S354, andremoving step S355.

The color fringing removing unit 350 sets the R plane as a removaltarget, and uses the G plane as a reference plane.

In spatial calculation step S351, intensity gradient maps Rlea and Gleafor the R and G planes are generated by considering each pixel in turn.

For each considered pixel in the R plane at location (x,y) a gradientvector Rlea is calculated based on the intensity values of neighbouringpixels in the R plane.

Similarly, for the same considered pixel in the G plane (i.e. the pixelat location (x,y) or the location closest thereto in the G plane) agradient vector Glea is calculated based on the intensity values ofneighbouring pixels in the G plane.

For example,

$\begin{matrix}\begin{matrix}{{Rlea} = \left( {\frac{R}{x},\frac{R}{y}} \right)} \\{\equiv \left( {\frac{{R\left( {{x + 1},y} \right)} - {R\left( {{x - 1},y} \right)}}{2},\frac{{R\left( {x,{y + 1}} \right)} - {R\left( {x,{y - 1}} \right)}}{2\;}} \right)} \\{{Glea} = \left( {\frac{G}{x},\frac{G}{y}} \right)} \\{\equiv \left( {\frac{{G\left( {{x + 1},y} \right)} - {G\left( {{x - 1},y} \right)}}{2},\frac{{G\left( {x,{y + 1}} \right)} - {G\left( {x,{y - 1}} \right)}}{2}} \right)}\end{matrix} & (4)\end{matrix}$

where R(x+1,y) and G(x+1,y) are the intensity values of pixels on theright side of the considered pixels in the R and G planes,

R(x−1,y) and G(x−1,y) are the intensity values of pixels on the leftside of the considered pixels in the R and G planes, R(x,y+1) andG(x,y+1) are the intensity values of pixels immediately below theconsidered pixels in the R and G planes, and R(x,y−1) and G(x,y−1) arethe pixel values of pixels immediately above the considered pixels inthe R and G planes.

In the estimation step S353, an estimated value of the degree of colorfringing in the R plane is generated for each pixel of the color image.The estimation method changes depending on whether the pixel intensityvalue R is saturated. Considering the two cases where R is saturated andis not saturated, two estimated amounts E1 and E2 are calculated in S353a and S353 b.

FIG. 20 shows the typical intensity profile of red fringing.

In FIG. 20, the abscissa axis represents a section on an image (pixelposition), and the ordinate axis represents the intensities of the R andG planes. In FIG. 20, a bright object exceeding the saturationbrightness exists at the center. The bottom of the profile exponentiallyspreads at the periphery, which is not bright originally, of the lightsource by light leaking from the light source owing to aberration orflare. Even the G plane suffers fringing, and its profile spreads to acertain degree though the spread is smaller than that of the R plane. Anintensity at a predetermined saturation level or higher cannot bemeasured by the image sensor, and is rounded down. If the R intensityexceeds the G intensity in these profiles, red fringing occurs.

From this, the third embodiment estimates the R fringing amount from thegradient of the R brightness profile. In S353 a, a first estimatedfringing amount E1 is calculated by multiplying the absolute value ofthe R gradient vector Rlea by a coefficient k1:

E1=k1|Rlea|

where k1 is a positive value and is preferably around 3.

However, the brightness gradient is 0 in the region A1 where R issaturated, and a brightness gradient before saturation cannot beobtained. In S353 b, therefore, a second estimated fringing amount E2 iscalculated for this region A1. In S353 b, the estimated fringing amountE2 is determined from the G gradient Glea:

E2=k2|Glea|

where k2 is a positive value and is preferably around 3.

In region determination step S352, the degree S of saturation iscalculated by executing non-linear conversion for the intensity of the Rplane. The non-linear conversion represents whether R is saturated. Thedegree S of saturation is 1 in a region where the R intensity issaturated and 0 in a region where the R intensity is not saturated. Smay take one of the two values 0 and 1, or alternatively S may take avalue which continuously changes from 0 to 1, as shown in FIG. 21. Oneor the other of the first an second estimated values E1 and E2calculated in S353 is selected in accordance with the degree S ofsaturation. In the case in which S takes one of the two values 0 and 1,a new estimated amount E is set to

E=E1 (for S=0)

E=E2 (for S=1)

In the alternative case in which S takes a value which continuouslychanges from 0 to 1, the new estimated amount E is set to

E=(1−S)E1+S·E2

In excessive removal suppressing step S354, the estimated amount E iscorrected to obtain an actual removal amount E′ for the R plane. In thisstep, change of the hue H upon removal is limited within a predeterminedangle δ in this embodiment. The hue-chroma plane is as shown in FIG. 22.When the estimated amount E is removed from the R intensity, thehue-chroma plane moves down as indicated by dotted arrows. To limitchange of the hue within a predetermined angle δ, a an original hue Horibefore removal is obtained first:

Hori=H(R,G,B)

Then, negative and positive R removal amounts En and Ep for changing thehue angle negatively (clockwise in FIG. 22) and positively(anticlockwise) by the predetermined angle δ are calculated such thatH(R−En, G, B)=Hori−δ

H(R−Ep,G,B)=Hori+6

δ is preferably set to about 10° to 45°.

From En, Ep, and region determination step S352, the removal amount E′is set to

E′=min(E,max(En,Ep,0))

Then, E′ is transferred to removing step S355.

In FIG. 22, E′=En.

In removing step S355, a new intensity of the R plane is calculated bysubtracting the removal amount E′ from the intensity of the R plane:

R=R−E′

A color image whose R plane is corrected is transferred as an outputfrom the color fringing removing unit 350 to the vision correction unit160.

In the third embodiment, only upper, lower, right, and left pixelsadjacent to a pixel subjected to removal are referred to in colorfringing removal. Thus, no large-capacity frame memory is necessary, anda buffer memory for two lines suffices to process the entire image byraster scanning. An image processing apparatus can be implemented as ahigh-speed, compact circuit.

As described above, the third embodiment can effectively remove colorfringing by image processing.

However, if fringing is completely removed, the brightness and color ofa bright object exceeding the saturation brightness cannot beidentified. To prevent this, the above-described embodiments set areference plane to reduce conspicuous fringing of a color plane to thesame level as that of the reference plane. The embodiments do not aim toreduce fringing much more. Even an image having undergone color fringingremoving processing has a certain degree of fringing, and allowsidentifying the brightness and color of a bright object.

Because of the image processing provided by embodiments of the presentinvention, it is sufficient for the imaging optical system that is usedto capture the color image capturing apparatus to remove aberration inat least one color band. Other aberration corrections, downsizing, andcost reduction demanded of the imaging optical system can be achieved ona higher level.

Other Embodiments

An embodiment of the present invention can also be implemented usingsoftware. Accordingly, a further aspect of the present invention canprovide a program which, when run on a computer or processor, causes thecomputer or processor to execute an image processing method according toany of the preceding embodiment. Such a program can be provided byitself or can be carried in or on a carrier medium. The carrier mediummay be a transmission medium, for example a signal such as a downloadsignal transmitted via a network. The carrier medium may also be astorage medium such as a disk or memory stick. For example, a storagemedium (or recording medium) which stores software program codes forimplementing the functions of the above-described embodiments may besupplied to the system or apparatus. The computer (or processor such asa CPU or MPU) of the system or apparatus reads out and executes theprogram codes stored in the storage medium. In this case, the programcodes read out from the storage medium implement the functions of theabove-described embodiments by themselves, and the storage medium whichstores the program codes constitutes the present invention. In additionto the case where the functions of the above-described embodiments areimplemented when the computer executes the readout program codes, thepresent invention incorporates the following case. That is, thefunctions of the above-described embodiments are implemented when theoperating system (OS) or the like running on the computer performs partor all of actual processing based on the instructions of the programcodes.

The present invention also incorporates the following case. That is, theprogram codes read out from the storage medium are written in the memoryof a function expansion card inserted into the computer or a functionexpansion unit connected to the computer. After that, the functions ofthe above-described embodiments are implemented when the CPU of thefunction expansion card or function expansion unit performs part or allof actual processing based on the instructions of the program codes.

When the present invention is applied to the storage medium, the storagemedium stores program codes corresponding to the above-describedprocedures.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application Nos.2006-332610 and 2006-332611 both filed Dec. 8, 2006, which are herebyincorporated by reference herein in their entirety.

1. An image processing apparatus, for processing original image datarepresenting a captured color image that has been subjected tophotoelectric conversion, comprising: estimation unit which generatesestimated color fringing information representing an estimate of colorfringing in the captured image on the basis of the original image data;and removing unit which employs the estimated color fringing informationto generate modified image data based on the original image data so asto compensate for such color fringing in at least a part of the capturedimage.
 2. The apparatus according to claim 1, wherein said estimationunit estimates the degree of color fringing in each region of the colorimage in accordance with a difference between signal intensities of theplurality of color planes.
 3. The apparatus according to claim 2,wherein said estimation unit uses, as a reference plane, a color planeother than a color plane from which said removing unit removes colorfringing, and estimates the degree of color fringing in each region ofthe color image in accordance with a difference between a signalintensity of the color plane subjected to removal and the referenceplane.
 4. The apparatus according to claim 1, wherein said estimationunit estimates the degree of color fringing in each region of the colorimage in accordance with a difference between a signal intensity of aluminance plane and a signal intensity of a color difference plane inshooting the color image.
 5. The apparatus according to claim 3, whereinin a region where the color plane subjected to removal is saturated,said estimation unit outputs a value for increasing the degree of colorfringing in a direction away from a region where the reference plane issaturated.
 6. The apparatus according to claim 3, wherein in a regionwhere the color plane from which said removing unit removes colorfringing is not saturated, said estimation unit outputs a value fordecreasing the degree of color fringing in a direction away from aregion where the color plane subjected to removal is saturated.
 7. Theapparatus according to claim 5, wherein a ratio at which the degree ofcolor fringing is changed changes depending on an image height of anobject image.
 8. The apparatus according to claim 5, wherein a ratio atwhich the degree of color fringing is changed changes depending on astate of an imaging optical system which forms an object image.
 9. Theapparatus according to claim 3, wherein said estimation unit outputs avalue for increasing the degree of color fringing as a differencebetween a saturation radius of the color plane subjected to removal anda saturation radius of the reference plane increases.
 10. The apparatusaccording to claim 1, further comprising a spatial calculation unitwhich calculates a distance to a region where color fringing is removed,from a saturated region where charges in an image sensing element forphoto-electrically converting an object image are saturated, whereinsaid estimation unit estimates the degree of color fringing in eachregion of the color image by using the calculated distance.
 11. Theapparatus according to claim 1, further comprising a spatial calculationunit which uses, as a reference plane, a color plane other than a colorplane from which said removing unit removes color fringing, and performsnon-linear conversion for a signal intensity of the reference plane,wherein said estimation unit estimates the degree of color fringing ineach region of the color image by using a non-linear conversion resultof said spatial calculation unit.
 12. The apparatus according to claim11, wherein the non-linear conversion includes conversion into twovalues representing whether an intensity of the reference plane issaturated.
 13. The apparatus according to claim 11, wherein said spatialcalculation unit performs convolution processing for a value havingundergone the non-linear conversion, and a kernel of the convolutionprocessing envelops a plurality of point-spread functions of an imagingoptical system that change depending on an image location or a state ofthe imaging optical system in shooting.
 14. The apparatus according toclaim 1, further comprising a spatial calculation unit which uses, as areference plane, a color plane other than a color plane from which saidremoving unit removes color fringing, and calculates a gradient of asignal intensity of a color plane, wherein said estimation unitestimates the degree of color fringing in each region of the color imageby using at least either of a gradient of a signal intensity of thecolor plane subjected to removal and a gradient of a signal intensity ofthe reference plane.
 15. The apparatus according to claim 14, furthercomprising a region determination unit which determines whether thecolor plane from which said removing unit removes color fringing issaturated, wherein said estimation unit selects either of the gradientof the signal intensity of the color plane subjected to removal and thegradient of the signal intensity of the reference plane on the basis ofa determination result of said region determination unit.
 16. Theapparatus according to claim 1, further comprising a calculation unitwhich calculates a corrected estimated value by correcting the estimatedvalue of the degree of color fringing estimated by said estimation unit,in order to suppress excessive removal of a color fringing componentfrom the color image when removing the color fringing component of thecolor image from the color image, wherein said removing unit subtracts,from the color image, the corrected estimated value calculated by saidcalculation unit.
 17. The apparatus according to claim 16, wherein saidcalculation unit calculates the corrected estimated value so as tosuppress change of hue of the color image that is caused by removal bysaid removing unit.
 18. A method of processing original image datarepresenting a captured color image that has been subjected tophotoelectric conversion, the method comprising the steps of: generatingestimated color fringing information representing an estimate of colorfringing in the captured color image on the basis of the original imagedata; and employing the estimated color fringing information to generatemodified image data based on the original image data so as to compensatefor such color fringing in at least a part of the captured image. 19.The method according to claim 18, further comprising a calculation stepof calculating a corrected estimated value by correcting the estimatedvalue of the degree of color fringing estimated in the estimation step,in order to suppress excessive removal of a color fringing componentfrom the color image when removing the color fringing component of thecolor image from the color image, wherein in the removing step, thecorrected estimated value calculated in the calculation step issubtracted from the color image.
 20. A program which causes a computerto execute an image processing method defined in claim 18.